Papers with Transformer architectures
Word Acquisition in Neural Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Language models acquire individual words during training, based on unigram token frequencies, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. |
| Approach: | They examine how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory. |
| Outcome: | The models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. |
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |
LTRC-MT Simple & Effective Hindi-English Neural Machine Translation Systems at WAT 2019 (D19-52)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) is a promising approach for low resource languages. |
| Approach: | They propose to use both Recurrent Neural Networks & Transformer architectures to train NMT models. |
| Outcome: | The proposed model outperforms Statistical Machine Translation (SMT) techniques on a low resource Hindi-English language pair. |
ETC: Encoding Long and Structured Inputs in Transformers (2020.emnlp-main)
Copied to clipboard
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang
| Challenge: | Existing models for natural language processing (NLP) have been challenging to scale attention to longer inputs. |
| Approach: | They propose an extended Transformer construction architecture that scales attention to longer inputs by combining global-local attention with relative position encodings and a "Contrastive Predictive Coding" objective. |
| Outcome: | The proposed architecture scales attention to longer inputs and encodes structured inputs. |
Learning Language Specific Sub-network for Multilingual Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Multilingual neural machine translation models suffer from performance degradation when learning multiple languages. |
| Approach: | They propose to use LaSS to jointly train a single unified multilingual MT model. |
| Outcome: | The proposed model gains on 36 language pairs by up to 1.2 BLEU and zero-shot translation with 8.3 BLUE on 30 language pairs. |
Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations (D19-3)
Copied to clipboard
| Challenge: | Currently, the main alternatives to deal with sequences are Recurrent Neural Networks (RNN) architectures and the Transformer. |
| Approach: | They propose a web-based tool that visualizes the sentence and token representations of RNNs and Transformer architectures at the sentence level. |
| Outcome: | The proposed visualization tool analyses gender inequalities in contextual word embeddings and the common language representation in a multilingual machine translation system. |
EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks . alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructures. |
| Approach: | They propose a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. |
| Outcome: | The proposed solution achieves comparable performance to baseline Transformer-based LLMs while optimizing memory consumption and time to first token. |
Fact Checking with Insufficient Evidence (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on how to automate fact checking relies on information obtained from external sources. |
| Approach: | They propose a fluency-preserving method for omitting information from the evidence at the constituent and sentence level and a diagnostic dataset for FC with omitted evidence. |
| Outcome: | The proposed method improves evidence sufficiency prediction by 17.8 F1 score and 2.6 F1 scores. |
Scheduled Sampling for Transformers (P19-2)
Copied to clipboard
| Challenge: | Existing studies show that scheduled sampling can be applied to recurrent neural networks to avoid exposure bias. |
| Approach: | They propose to use teacher forced embeddings and model predictions to avoid exposure bias in sequence-to-sequence generation. |
| Outcome: | The proposed technique achieves performance close to a teacher-forcing baseline on two language pairs and is promising for future research. |
Transition-based Parsing with Stack-Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing parsing systems use local or global models of the parser state to improve performance. |
| Approach: | They propose to modify the sequence-to-sequence Transformer to model global or local parser states in transition-based parsing. |
| Outcome: | The proposed model significantly improves performance on dependency and Abstract Meaning Representation (AMR) parsing tasks. |
Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines whether vision-and-language models learn syntactic dependencies . a controlled evaluation of the models is crucial for a precise and rigorous test of their knowledge . |
| Approach: | They propose a task to evaluate understanding of predicate-noun dependencies in a controlled setup. |
| Outcome: | This study compares state-of-the-art models with a case study on predicate-noun dependencies. |
Formality Style Transfer with Shared Latent Space (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches for formality style transfer use neural networks for sentence generation, but the dataset for formal style transfer is considerably smaller than translation corpora. |
| Approach: | They propose a new approach for formality style transfer using shared latent space and two auxiliary losses. |
| Outcome: | The proposed approach outperforms baselines in various settings, especially when limited data is available. |
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures. |
| Approach: | They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. |
| Outcome: | The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model. |
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show draft-then-verify paradigm can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding. |
| Approach: | They propose to use Spec-Drafter and Spec Verification to accelerate autoregressive (AR) decoding by combining a model optimized for efficient and accurate drafting and a reliable method for verifying the drafted tokens efficiently. |
| Outcome: | The proposed method achieves 5x speedup on seq2seq tasks with comparable generation quality to beam search decoding, refreshing the impression that draft-then-verify paradigm introduces only 1.4x2x speed up. |
Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Modern neural language models achieve high accuracy in text generation, yet precise control over generation length remains underdeveloped. |
| Approach: | They propose a method to provide robust length control using Reverse Positional Embeddings. |
| Outcome: | The proposed method provides stable length fidelity without degrading text accuracy . the proposed method generalizes well to unseen target lengths . |
SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | SupCL-Seq extends contrastive learning from computer vision to sequence classification tasks. |
| Approach: | They propose a supervised alternative to Masked Language Modeling (MLM) that extends contrastive learning to sequence optimization in NLP by altering the dropout mask probability in standard Transformer architectures. |
| Outcome: | The proposed method leads to large gains on the GLUE benchmark, including 6% absolute improvement on CoLA, 5.4% on MRPC, 4.7% on RTE and 2.6% on STS-B. |
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained Transformers are remarkably robust to pruning. |
| Approach: | They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance. |
| Outcome: | The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance. |
COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent debates have shown that attention maps and attribution methods are unreliable. |
| Approach: | They propose a model-agnostic XAI technique that generates meaningful explanations from the last layer of a neural net model trained on an NLP classification task by using Non-Negative Matrix Factorization to discover concepts the model leverages to make predictions. |
| Outcome: | The proposed technique generates meaningful explanations from the last layer of a neural net model trained on an NLP classification task without compromising the accuracy of the underlying model or requiring a new one to be trained. |
When depth is redundant: Efficient transformer-based speech anti-spoofing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing anti-spoofing countermeasures exhibit limited generalization to unseen spoof attacks, especially in out-of-domain evaluation settings. |
| Approach: | They propose a training strategy that aligns shallow and intermediate representations with those of the final transformer layer for speech deepfake detection. |
| Outcome: | The proposed model improves robustness to unseen spoofing attacks and enhances out-of-domain generalization over strong baselines. |
Probing for Referential Information in Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Neural network-based language models (LMs) have been shown to learn relevant properties of language without being explicitly trained for them. |
| Approach: | They extend their previous work to analyze whether language models capture anaphoric relations and pronoun-antecedent relations in English. |
| Outcome: | The Transformer outperforms the LSTM in all analyses. |
Analysis of Tree-Structured Architectures for Code Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Code generation is the task of generating code snippets from input user specifications written in natural language (NL). |
| Approach: | They evaluate the significance of input parse trees for code generation by using constituency-based parsers as input and an abstract syntax tree as the target. |
| Outcome: | The proposed models on a Python-based code generation dataset and a semantic parsing dataset show that constituency trees encoded using a structure-aware model improve performance. |
ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks. |
| Approach: | They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy. |
| Outcome: | The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks. |
A Closer Look at Parameter Contributions When Training Neural Language and Translation Models (2022.coling-1)
Copied to clipboard
| Challenge: | Neural models and Transformers have been used for almost every NLP task . however, the intrinsic dynamics of the training procedure have not been studied in depth for highly complex network architectures. |
| Approach: | They analyze the learning dynamics of neural language and translation models using Loss Change Allocation indicator . they use a standard Transformer architecture to train a model with three learning objectives . |
| Outcome: | The proposed model is based on a standard model that is used for training tasks. |
ABC: Attention with Bounded-memory Control (2022.acl-long)
Copied to clipboard
Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, Noah A. Smith
| Challenge: | Existing approaches to attention with bounded-memory control (ABC) have a quadratic complexity in sequence lengths, making it prohibitive for long sequences. |
| Approach: | They propose a new abstraction that bounds memory size to improve efficiency . they propose bounded-memory control, which connects several efficient attention variants . |
| Outcome: | The proposed approach outperforms existing approaches on language modeling, machine translation, and masked language model finetuning. |
Datasets for Multilingual Answer Sentence Selection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for Answer Sentence Selection (AS2) focus on English due to the scarcity of annotated datasets. |
| Approach: | They propose to use a large language model to train answer sentences in English . they use annotated datasets from English and other languages to train AS2 models . |
| Outcome: | The proposed datasets are highly performant and close the performance gap between English and other languages. |
On the In-context Generation of Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have the ability of in-context generation (ICG) when given an in-text prompt, they can implicitly recognize the pattern of the examples and complete the prompt in the desired way. |
| Approach: | They propose a plausible latent variable model to model the distribution of pretrained corpora and formalize ICG as a problem of next topic prediction. |
| Outcome: | The proposed model can model the distribution of pretrained corpora and then formalize ICG as a problem of next topic prediction. |
Low-resource Neural Machine Translation: Benchmarking State-of-the-art Transformer for Wolof<->French (2022.lrec-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) systems can translate between French (FR) 1 and Wolof (WO, ISO 639-3), a lowresource Niger-Congo language mainly spoken in Senegal (Gamble, 1950). |
| Approach: | They propose two neural machine translation systems based on sequence-to-sequence with attention and Transformer architectures to translate between French (FR) 1 and Wolof (WO, ISO 639-3). |
| Outcome: | The proposed models outperform the classic sequence-to-sequence model in all settings and are less sensitive to noise. |
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful Transformer architectures produce dull high-frequency phrases, severely hurting the diversity and novelty of generated text. |
| Approach: | They propose a method to control the sharpness of the attention distribution by python code and use it to learn a Bayesian approximation of posterior attention. |
| Outcome: | The proposed method improves diversity and novelty while maintaining comparable quality on conditional and unconditional generation tasks. |
Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for topic-controllable summarization are limited by their recurrent architectures and require modifications to the model's architecture for controlling the topic. |
| Approach: | They propose a new topic-oriented evaluation measure to automatically evaluate the generated summaries based on the topic affinity between the generated summary and the desired topic. |
| Outcome: | The proposed method achieves better performance compared to more complicated embedding-based approaches while also being significantly faster. |